1. Field of the Invention
The invention relates generally to the field of digital image processing and digital image understanding, and more particular to a system for determining the orientation in photographic and other similar images containing sky.
2. Description of the Related Art
Digital images may be acquired through a number of means, including scanning a picture captured on negative films or direct downloading from a digital camera. A photographer can take a picture while holding the camera in any of the possible ways (upright, upside down, rightside up, or leftside up). In addition to that, the film may be loaded from left to right or from right to left, resulting in unpredictable image orientation. It is beneficial to have an automatic way of sensing image orientation to avoid tedious reorientation of an image, which would otherwise need to be performed by an operator. It is also an efficiency issue for mass image processing in commercial systems.
Sky is among the most important subject matters frequently seen in photographic images. Detection of sky can often facilitate a variety of image orientation, understanding, enhancement, and manipulation tasks. Sky is a strong indicator of an outdoor image for scene categorization (e.g., outdoor scenes vs. indoor scenes, picnic scenes vs. meeting scenes, city vs. landscape, etc.). See, for example M. Szummer and R. W. Picard, xe2x80x9cIndoor-Outdoor Image Classification,xe2x80x9d in Proc. IEEE Intl. Workshop on Content-based Access of Image and Video Database, 1998 and A. Vailaya, A. Jain, and H. J. Zhang, xe2x80x9cOn Image Classification: City Vs. Landscape,xe2x80x9d in Proc. IEEE Intl. Workshop on Content-based Access of Image and Video Database, 1998 (both of which are incorporated herein by reference).
The most prominent characteristic of sky is its color, which is usually light blue when the sky is clear. Such a characteristic has been used to detect sky in images. For example, U.S. Pat. No. 5,889,578, entitled xe2x80x9cMethod And Apparatus For Using Film Scanning Information To Determine The Type And Category Of An Imagexe2x80x9d by F. S. Jamzadeh, (which is incorporated herein by reference) mentions the use of color cue (xe2x80x9clight bluexe2x80x99xe2x80x9d) to detect sky without providing further description.
In a work by Saber et al. (E. Saber, A. M. Tekalp, R. Eschbach, and K. Knox, xe2x80x9cAutomatic Image Annotation Using Adaptive Color Classification,xe2x80x9d CVGIP: Graphical Models and Image Processing, vol. 58, pp. 115-126, 1996, incorporated herein by reference), color classification was used to detect sky. The sky pixels are assumed to follow a 2D Gaussian probability density function (PDF). Therefore, a metric similar to the Mahalonobis distance is used, along with an adaptively determined threshold for a given image, to determine sky pixels. Finally, information regarding the presence of sky, grass, and skin, which are extracted from the image based solely on the above-mentioned color classification, are used to determine the categorization and annotation of an image (e.g., xe2x80x9coutdoorxe2x80x99xe2x80x9d, xe2x80x9cpeoplexe2x80x9d). Note that Saber does not deal with determination of image orientation, which is assumed known.
U.S. Pat. No. 5,642,443, entitled, xe2x80x9cWhole Order Orientation Method And Apparatusxe2x80x9d by Robert M. Goodwin, (which is incorporated herein by reference) uses color and (lack of) texture to indicate pixels associated with sky in the image. In particular, partitioning by chromaticity domain into sectors is utilized by Goodwin. Pixels with sampling zones along the two long sides of a non-oriented image are examined. If an asymmetric distribution of sky colors is found, the orientation of the image is estimated. The orientation of a whole order of photos is determined based on estimates for individual images in the order. For the whole order orientation method in Goodwin to be successful, a sufficiently large group of characteristics (so that one with at least an 80% success rate is found in nearly every image), or a smaller group of characteristics (with greater than a 90% success ratexe2x80x94which characteristics can be found in about 40% of all images) is needed. Therefore, with Goodwin, a very robust sky detection method for any given image is not required.
For recognizing the orientation of an image, knowledge of sky and its orientation may indicate the image orientation for outdoor images. Contrary to the common belief, a sky region is not always at the top of an image. In many cases, a sky region may touch more than the top border of an image.
A major drawback of conventional techniques is that they utilized the location of objects (e.g., sky) to determine orientation. Those conventional systems which identify sky in images suffer from the disadvantage that the sky may not always appear at the top of the image and may also touch more than the top border of an image. Further, conventional techniques cannot differentiate sky from other similarly colored or textured subject matters, such as a blue wall, a body of water, a blue shirt, and so on. Failure to reliably detect the presence of sky, in particular false positive detection, and an inadequate assumption that the location of the sky indicates the image orientation, may lead to failures in orientation process.
The invention provides a robust image orientation process that identifies sky based on color hue classification, texture analysis, and physics-motivated sky trace analysis. The invention utilizes hue color information to select bright, sky colored pixels and utilizes connected component analysis to find potential sky regions. The invention also utilizes gradient to confirm that sky regions are low in texture content and segments open space, defined as smooth expanses, to break up adjacent regions with similar sky color beliefs but dissimilar sky colors. The invention further utilizes gradient to determine the zenith-horizon direction and uses a physics-motivated sky trace signature to confirm the determined orientation indeed comes from a valid sky region (as opposed to a merely xe2x80x9csky-coloredxe2x80x9d region).
More specifically, the invention can take the form of a method, image recognition system, computer program, etc., for determining image orientation. The invention classifies potential sky pixels in the image by color, identifies spatially contiguous regions of the potential sky pixels, identifies actual sky regions by eliminating ones of the spatially contiguous regions that have a texture above a predetermined texture threshold, computes desaturation gradients of the actual sky regions, classifies the image as one of portrait and landscape based on average horizontal and vertical desaturation gradient values of pixels within each of the actual sky regions, determines orientation of the image based on a polarity of the average horizontal and vertical desaturation gradients, and confirms that the actual sky regions are true sky regions by comparing the desaturation gradients with a predetermined desaturation gradient for sky.
The invention also maintains only ones of the spatially contiguous regions as the actual sky regions which have substantially dissimilar average horizontal and vertical gradients, have a color distribution which is consistent with a predetermined color distribution for sky, and contact a border of the image.
The classifying of the image as one of portrait and landscape includes classifying the image as the portrait if an absolute value of the average horizontal desaturation gradient is greater than an absolute value of the average vertical desaturation gradient and classifying the image as the landscape if an absolute value of the average horizontal desaturation gradient is less than an absolute value of the average vertical desaturation gradient. The desaturation gradients comprise desaturation gradients for red, green and blue trace components of the image.
The predetermined desaturation gradient for sky comprises, from horizon to zenith, a decrease in red and green light trace components and a substantially constant blue light trace component.
The classifying potential sky pixels in the image by color comprises forming a belief map of sky color pixels, computing an adaptive threshold of sky color, and determining if the potential sky pixels exceed the threshold. The computing of the adaptive threshold comprises identifying a first valley in a belief histogram derived from the belief map. The belief map and the belief histogram are unique to the image.
One advantage of the invention lies in the utilization of a physical model of the sky based on the scattering of light by small particles in the air to determine image orientation. By using a physical model (as opposed to a color or texture model), the invention is not likely to be fooled by other similarly colored subject matters such as bodies of water, walls, toys, and clothing. Further, the inventive region extraction process automatically determines an appropriate threshold for the sky color belief map. By utilizing the physical model in combination with color and texture filters, the invention produces results that are superior to conventional systems.